
L 

Number 


Hits 


Search Text 


DB 


Time stamp 


1 


168 


test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:36 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$. eels . 


IBM TDB 




2 


107 


(test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:54 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group 


IBM TDB 




3 


13 


( (test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:39 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 identif$7 






5 


13 


(((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:21 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBMJTDB 








group near3 identif$7) and sequenc$2 






8 


8 


((((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:41 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 identif$7) and sequenc$2) and 










instanc$ 6 






9 


5 


((((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:39 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 identif$7) and sequenc$2) and 










instanc$6 same object 






10 


25 


((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:40 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 (identif$7 or name or id or 










number) 






11 


24 


( (test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:55 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 (identif$7 or name or id or 










number) and sequence 






12 


8 


( ( (test$3 near3 (program or software or 


US PAT ; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:42 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 (identif$7 or name or id or 










number) and sequence) and instance same 










object 






13 


16 


(((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


12:42 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 (identif$7 or name or id or 










number) and sequence) and instance 






14 


5 


(((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:20 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near 3 (identif$7 or name or id or 










number) and sequence) and test$3 near3 










class 






15 


10 


((test$3 near3 (program or software or 


US PAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


13:28 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM TDB 








test near3 class 







Search History 9/3/03 5:57:26 PM Page 1 
C: \APPS\EAST\workspaces\09783250.wsp 




16 


23 


{((test$3 near3 (program or software or 


USPAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:21 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBMJTDB 








group near3 (identif$7 or name or id or 










number) and sequence) and (hierarch$6 or 










level or dependenc$4 or tree) 






17 


14 


(((test$3 near3 (program or software or 


USPAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:22 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM_TDB 








group near3 (identif$7 or name or id or 










number) and sequence) and (hierarch$6) 






18 


13 


((test$3 near3 (program or software or 


USPAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:55 






(population or group or modul$5) and 


DERWENT ; 








sequenc$2 and 717/$.ccls.) and group) and 


IBMJTDB 








java and database 






19 


8 


(((test$3 near3 (program or software or 


USPAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15:55 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBMJTDB 








group near3 (identif$7 or name or id or 










number) and sequence) and java 






20 


4 


( ( ( ( (test$3 near3 (program or software or 


USPAT; 


2003/09/03 






instruction or code) and test same 


EPO; JPO; 


15 : 55 






(population or group or modul$5) and 


DERWENT; 








sequenc$2 and 717/$.ccls.) and group) and 


IBM TDB 








group near3 identif$7) and sequenc$2) and 










instanc$6 same object) and database 







Search History 9/3/03 5:57:26 PM Page 2 
C: \APPS\EAST\workspaces\09783250 .wsp 



United States Patent [i<>] 

Rodrigues et al. 



US006067639A 
[n] Patent Number: 6,067,639 
[45] Date of Patent: *May 23, 2000 



[54] METHOD FOR INTEGRATING AUTOMATED 
SOFTWARE TESTING WITH SOFTWARE 
DEVELOPMENT 

[75] Inventors: James Perry Rodrigues, Kirkland; 

Orville Jay Potter, IV, Redmond, both 
of Wash. 

[73] Assignee: Microsoft Corporation, Redmond, 
Wash. 

[ * ] Notice: This patent issued on a continued pros- 
ecution application filed under 37 CFR 
1.53(d), and is subject to the twenty year 
patent term provisions of 35 U.S.C. 
154(a)(2). 



[21] 

[22] 

[51] 
[52] 
[58] 



[56] 



Appl. No.: 08/552,483 



Filed: 



Nov. 9, 1995 



Int. CI. 7 H02H 3/05; H03K 19/003 

U.S. CI 714/38; 395/701 

Field of Search 395/183.14, 701, 

395/704, 705, 500, 702; 364/267, 267.91; 

714/38 

References Cited 
U.S. PATENT DOCUMENTS 



5,450,586 9/1995 Kiizara et al 395/700 

5,475,843 12/1995 Halviatti et al 395/704 

5,485,615 1/1996 Wennmyr 395/702 

5,490,249 2/1996 Miller 395/183.14 

5,513,316 4/1996 Rodrigues et al 395/183.14 



5,522,073 5A996 Courant et al 395/701 

5,615,333 3A997 Juettoer et al 395/183.14 

5,671,415 9A997 Hossain 395/701 

5,751,941 5A998 Hinds et al 714/38 

Primary Examiner— Dieu-Minh T. Le 

Attorney, Agent, or Firm — Banner & Witcoff, Ud. 

[57] ABSTRACT 

A computer operable method for integrating and automating 
test procedures within a computer application program. 
Instantiated test operation objects of an object class defined 
by the present invention correspond to functions to be tested 
within the computer application program. The test operation 
objects are instantiated by calls to functions in a test opera- 
tion runtime library (DLL). The test operation objects 
include interface method functions which execute the 
intended test operation and simulate any required data, file, 
or memory I/O. Other aspects of the methods of the present 
invention provide rules which permit decisions as to the 
applicability of each test operation given the state of the 
application program or the context of the test operation 
sequence. The various test operation objects are selected to 
perform a sequence of test steps. In one mode of operation, 
the methods of the present invention randomly select among 
all the instantiated test operation objects. In another mode of 
operation, the methods of the present invention "playback" 
a previously recorded sequence of test operation objects to 
permit reproduction of failures in previous test sequences. In 
a third mode of operation, the methods of the present 
invention permit a user to modify or create "playback" files 
to customize a test case for a particular focus. 
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METHOD FOR INTEGRATING AUTOMATED 
SOFTWARE TESTING WITH SOFTWARE 
DEVELOPMENT 

FIELD OF THE INVENTION 5 

The present invention relates to the field of computer 
software development and testing, and in particular, to a 
method for integrating automated software testing into a 
product as it is being developed to thereby simplify subse- 
quent software product testing. 10 

PROBLEM 

It has been common in the computer software arts that 
software product development was a distinct and separate 
process from software product testing. In the standard soft- 
ware product development lifecycle, developmeot engineers 
would iteratively develop and refine a software product 
according to requirement or functional specifications. The 
development engineers frequently would test the product 2Q 
under development either in an ad-hoc manner or in a more 
rigorous manner. In either case, when the product was 
deemed to be completed by the development group of 
engineers, the software product was turned over to the 
testing process, usually a separate set of engineers from the ^ 
group that developed the software product. Most, if not all, 
of the testing process utilized in the development phase 
would be discarded and the test engineers would begin anew 
evaluating the software product as against its corresponding 
product specifications. 3Q 

In a typical test environment, the test engineers are 
provided with little or no internal information regarding the 
structure and design of the code comprising the software 
product. The testing effort would then proceed in a "black 
box" manner by applying test data (an external stimulus) and 35 
observing the output of the software product for proper 
response. In its crudest form, a test engineer determines 
("dreams up") potentially interesting test inputs and applies 
them to the software product while observing the output 
behavior of the software product for proper operation. This ^ 
form of manual testing permits wide flexibility for the test 
engineer in creating input stimuli but provides little assis- 
tance to the test engineer in reproducing problems found 
during the test sequence. Manual test requires the test 
engineer to carefully note the sequence of test inputs which 45 
led to a specific problem test result. 

Test engineers have developed or utilized a number of 
tools to assist in such "black box" testing. To automate the 
above discussed manual test process, scripting and macro 
languages are frequently used. The script/macro tool allows 50 
some degree of automation in the test sequence to aid in 
reproducing intermittent failures. A macro tool permits some 
added flexibility in the use of variables to customize the 
operation of the script based on externally supplied variable 
inputs. Some software products (e.g. word processing pro- 55 
grams or spreadsheet programs) offer built-in macro features 
to reproduce sequences of user input keystrokes. Other 
program user interface (UI) environments (such as the 
Xwindows programming environment) provide for redirec- 
tion of a program's input from a script/macro file and thus go 
may automate the testing of specific user keystroke 
sequences. 

Simple "BAT* files in the MS-DOS® environment or 
"shell scripts" in the UNIX® environment are exemplary of 
such scripting tools used to partially automate the software 65 
product testing process. Macro facilities in the "BAT* file or 
"shell script" interpreters and other macro replacement 



639 

2 

programs such as "m4" or "perl" in the UNIX® environment 
are exemplary of macro facilities used to enhance the 
scripting facilities in automating software product testing. 

However, such scripting/macro based testing tools still 
provide little or no assistance to the test engineer in observ- 
ing the software product output to automatically detect 
success and failure of each test. Collection and analysis of 
the test results remains largely a manual procedure. Corre- 
lation of the gathered test output with the timing and 
operation of the software product is difficult if not impos- 
sible. 

An additional problem with scripting/macro tools arises in 
that the tool itself as well as the test scripts themselves must 
be ported to each computing environment in which the 
software product is to be tested. For example, a particular 
operation in a PC based Microsoft Windows® application 
may be tested by simulating an "OPEN" menu operation 
while a similar function may require an w OK" menu opera- 
tion in an Apple Macintosh® computing environment. The 
test scripts would have to change to test the same function 
in these two exemplary computing environments. In other 
words, the testing of underlying functions of an application 
may have to change as the user interface changes over a 
variety of computing environments. This creates an addi- 
tional burden in porting and maintaining the test tools and 
test scripts along with the software products. In addition, the 
porting and maintenance of the scripting/macro tools and 
test cases can be a source of additional errors which may be 
erroneously attributed to the failure of the software product 
under test. 

Scripting/macro based test tools also tend to vary depend- 
ing upon the external stimuli needs of each software product 
under test. The user interface for the test engineer using 
"black box" automated test tools tends to vary as the needs 
of each software product vary. Test engineers therefore must 
potentially learn a new user interface mode of operation for 
each software product to be tested. 

All such "black box" testing methods, with or without the 
aid of automated scripting/macro testing tools, are typically 
performed without knowledge of the software product's 
internal code structure. This lack of structural knowledge 
can make testing more cumbersome and time consuming. 
The lack of structural knowledge precludes certain styles of 
testing which may focus on particular error prone aspects of 
the software product revealed only by knowledge of the 
software product's internal structure. Sometimes, for 
example, an error in one operation of the software product 
may not be externally observable in the output of the 
software product until subsequent operations are performed. 
The combination of the operations may eventually reveal the 
problem in an external manifestation, but the sequence of 
event may be lengthy back to the actual cause of the 
problem. Thus the use of external stimuli ("black box") test 
methods, even in association with various automated test 
tools, does not permit the test engineer to exercise judge- 
ment with respect to problems more easily located through 
testing with knowledge of the software product's internal 
code structure. 

In addition, as software products evolve with ever increas- 
ing complexity, the burden of testing is being shifted more 
toward the development teams. It is simply impossible with 
"black box" testing techniques to exhaustively test all pos- 
sible inputs (external stimuli) of many software products 
regardless of the size of the testing staff. Even testing of a 
large portion of the possible inputs is a difficult task in many 
cases. Therefore, the software development/testing lifecycle 
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has begun to shift more of the testing efforts onto the operation object (of a predefined test API object class) 
responsibility of the development staff. For example, it is corresponding to each operation to be tested. The test 
more frequent now that "acceptance testing" (testing for operation object identifies an execute interface method 
fundamental functionality) is made a part of the responsi- (provided by the application programmer) to be invoked to 
bmty of the development staff. An added benefit of shifting 5 execute the desired operation. Other interface methods of the 
some test burden to the development group is realized in the test operation object identify functions which are to be 
knowledge of internal code structure of the software prod- performed to simulate the generation of input data and the 
uct. The code structure knowledge of the development group acceptance of output data in response to operation of the 
may be applied to focus the testing effort on test sequences execute interface method. For example, one interface 
and scenarios most likely to create problems. In addition, the 10 melhod provides functions to be invoked for the manual or 
developers may construct the test cases to detect the failure automatic generation of input data to test the operation, 
in the test sequence as early as possible. It is a problem for Another interface melhod of the test operation object pro- 
developers to create a standardized test interface for the vides Actions to be invoked in response to I/O requests of 
automated performance of test sequences in a manner that operation under test. Another interface method of the test 
permits easy reproduction of the failed test sequences. 15 operation object defines a set of rules to permit decisions to 

. . ' , ,. . , be made in the invocation of the test object. 

It is apparent from the above discussion that a need exists __ ... . , . , , . , . 

for an automated testing tool which permits a standardized ^ "PP^ation program is designed by the development 

interface for the generation and execution of test cases, e °e mee , rs wlh *» test °P! ratlon defined 45 P a " of 

which permits random sequences of testing, which permits *» ^ T.^T™',^ T^TfiST^M 

j • rrij** jl-i. . ?n compiled and linked with the test tool library (DLL) to 

easy reproduction of failed test sequences, and which auto- 20 4 . ^ , • i «• / ; *i a 

„ f /. _ ? ^ , , n provide the standard object class manipulation test methods 

matically senses the success or failure of each test step. r . . . . J _ , £ . r . 4 4 

7 of the present invention. By defining the test operation 

SOLUTION objects as discussed above, the application programmer has 

provided information (in the form of a collection of instan- 

The present invention solves the above identified prob- 25 tiated objects) useful to the test methods of the present 

lems and others to thereby advance the state of the useful invention to invoke each test operation and detect success or 

arts by providing an easy to use, coding standard and user failure of each tested operation. Since the test operations are 

interface for a software developer to integrate testing of a designed into the application program, and the library func- 

software product into the development of the product. The uoas which implement the test operation classes of the 

testing application program interface (test API) is used by an 30 present invention are available in a variety of computing 

application program developer to integrate a standardized environments, the application program may be more easily 

test interface into the software product in its development tested in many environments without the need to port unique 

phase. Use of the test API by the application program test tools to each new computing environment. The test tools 

invokes standard functions in a dynamic linked library can be readily available in any computing environment to 

(DLL) to initiate a standard test user interface when the 35 which the development staff chooses to port the application 

application program is run. The test tools are integrated into program. In addition, the test tools of the present invention, 

the application program and are therefore ported to any new unlike scripting and macro methods of the past, do not 

computing environment by simply porting the application require the development or testing teams to port the test 

program to that environment. The development engineer, cases between the various computing environments on 

possibly in cooperation with the test engineers, thereby 40 which the program is to be tested. The test tools of the 

designs the test capability intc^the application program from present invention are available for all platforms on which the 

its inception. These test hooks, combined with the standard, application program may be run. The test suites are recorded 

easy to use, graphical user interface of the test tools of the m a portable manner by the test tools of the present invention 

present invention, permit any engineer in the product devel- m so<alled playback files. 

opment team to test the product. Development engineers 4S application program is designed by the application 
may more effectively test the application program during its programmer to invoke the test tool at an appropriate stage in 
development phase to reduce the number of problems dis- me runtime startup or processing of the application program, 
covered later in the product's lifecycle. Test engineers may j^e test tool may then be operated in one of three modes: 
apply the same easy to use graphical interface to more namely, random automated test operation selection, play- 
thoroughly test the functionality of the product. 50 back automated test operation selection, or manual playback 

The test API functions in the DLL provide a consistent, creation mode, 
friendly, easy to use interface for testing of the application in the random automated test selection mode, the test 
program. The DLL test functions may be operated in a methods of the present invention pseudo randomly select 
random mode to generate pseudo-random test sequences. from among the set of instantiated test operation objects 
The DLL test functions automate recording of the pseudo- 55 defined by the application programmer. As each test opera- 
random seed and the corresponding sequence of test steps tion object is randomly selected, the execute function 
for easier reproduction of problems generated in the random method associated with the object is invoked to perform the 
sequence. The DLL functions can be used to automatically developer's designed test function. The random selections 
determine the success or failure of the corresponding test are logged in a playback file to permit later reproduction of 
step by sensing a return value from the invocation of a 60 a failed sequence. In addition, the results of each execute 
corresponding sequence of execution in the application function method invocation are logged in a file by the 
program. In addition, the application program may directly methods of the present invention so that a sequence may be 
invoke other DLL functions to record an application failure. reproduced if it produces a problem in the application 
This alternative method is referred to herein as the assertion program under test. This process continues randomly select- 
method. 65 m g test operation objects for which the corresponding 

The application programmer determines the operations to execute function is invoked until directed to stop by a user 

be tested in the application program and instantiates a test or other termination conditions are met. 
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In the playback automated test operation, the test opera- knowledge of the internal structure of the application pro- 

tion selections made in a previous test sequence are replayed gram. By contrast, it is common in present software devel- 

from the playback file created during an earlier performance opment and test environments that the quality assurance or 

of the test sequence. This mode of operation is used, for testing team operates using "black box" techniques 

example, to permit reproduction of test sequences which led 5 (applying stimuli and observing results without such knowl- 

to erroneous results in previous invocations. edge of the structure of the application program). In this 

The manual playback creation mode of operation permits W"»l software development and test environment, the 

a user of the present invention to create or edit a playback development team helps determine the requirements and 

file. This mode allows the user finer control over the test functional specification of the application program as 

sequencing to more rapidly debug an application program. 10 depicted in element 100. In element 102, the development 

Hie methods of the present invention enable automation team develo t f s a PP'°P™ le F"W ™* data structures to 
of the testing of an application program while integrating the ™Pl"nent the requirements and functional specifications. In 
test cases with the program development. This integration of conjunction with the development thereof, the development 
the test case and program development permits more thor- team ven f e A s "validates the program against the require- 
ough analysis and regression of the application program. 15 mems and ^tional specifications. In this typical applica- 
ble automation aspects of the test methods of the present * on PJ 0 S ram development process, the development team 
invention permit easier, unattended test operation to more has lhus completed its task and provides the application 
effectively test an application program. P ro S ram t0 ! he assurance or testing team for mde- 

pendent testing and verification of the application program. 

BRIEF DESCRIPTION OF THE DRAWINGS 20 In element 104, the quality assurance or testing team 

performs a regression test to verify that previously reported 

FIG. 1 is a flowchart depicting the lifecycle flow of an and fixed problems have not recurred in this new version of 

application program through development and test as pres- the application program. Next, at element 106, the quality 

ently known in the art; assurance or testing team performs new testing of the 

FIG. 2 is a flowchart depicting the lifecycle flow of an 25 application program to reveal potential new problems aris- 

application program through development and test as ing in the current version of the application program. Due to 

improved by the tools and methods of the present invention; the nature of the prior testing tools, the test team must 

FIG. 3 is a block diagram depicting the relationship initiate measures at element 108 (often manual) to document 

between script oriented external test tools and an application me ^P 8 required to reproduce the new problems discov- 

program as known in the art; 30 ere< *- 

• i| < j* j • . * , . . At element 110, the quality assurance or testing team 

FIG. 4 is a block diagram depicting the relationship . . ■ . . I . . . * 

. , • - * j * i * * < i j i • . ■ determines whether or not the number and seventy of 

between script oriented external test tools and an application . , ... . , J , 

*u , i j .i j p if, * revealed problems are too high to permit release of the 

program as improved by the tools and methods of the present . r. r . . . . 

invention- application program to potential users. If it is determined 

35 that the number and severity of problems revealed is not too 

FIG. 5 is a flowchart depicting the control flow of an high> lhe quahty assurance or tesling will release the 

application program with test operation objects embedded appUcation program l0 po tential ^ as depicled ^ element 

therein in accord with the present invention; 114 . If me number or severity of the problems revealed is too 

FIG. 6A is a flowchart showing additional detail of the high, the quahty assurance or testihgjeam does not release 

random test operation element of FIG. 5; 40 the application program to potential users^but rather reports 

FIG. 6B is a flowchart showing additional detail of the the open problems to the development team as depicted in 

random test operation of FIG. 5; element 112. In parallel with release of the application 

FIG. 7 is a flowchart showing additional detail of the program as depicted in element 114, the quality assurance or 

playback test operation of FIG. 5; testing team will also report the open problems to the 

FIG. 8 is a flowchart showing additional detail of the 45 35 shown b V ^menl :112. With the reporl t of open 

manual test operation of FIG. 5; problems returned to the development team, the develop- 

rt . ment team repeats its previous efforts in element 102 to 

FIG. 9 is a screen display showing an exemplary user re . d esign the application program to resolve problems 

interface which permits the user to define group and test reported by lhe quality assurance or lesting team . ^ cycle) 

operation weights for the random test operations of FIG. 5; 50 elements 102-112, repeats until the quahty assurance or 

FIG. 10 is a screen display showing an exemplary user testing team finds no remaining problems in the application 

interface which permits the user to define options for the program worthy of reporting to the development team, 

random test operations of FIG. 5; and FIG. 3 depicts a typical computing environment 308 in 

FIG. 11 is a screen display showing an exemplary user which the quality assurance or testing team performs their 

interface which permits the user to customize a playback test 55 testing function. A typical testing approach involves the use 

sequence for the playback test operations of FIG. 5. of a script oriented test tool 304 to apply external stimuli to 

appUcation program 306. The application program receives 

DETAILED DESCRIPTION OF THE the externally applied stimuli as a simulated user input and 

INVENTION generates an appropriate response by performing its desig- 

OVERVIEW — PRIOR ART 60 nated function on that provided input. The script oriented 

FIG. 1 describes the overall flow of a typical software test tool 304 detects the response generated by the applica- 

development and test life cycle for an application program tion program 306 to verify proper operation of the applica- 

product. A vertical dashed line in the middle of FIG. 1 tion program 306 in response to the applied stimuli. Script 

delineates tasks performed by a development team on the oriented test tool 304 may receive inputs from test suite or 

left side of the dashed line, and tasks performed by a quality 65 script files 300, which describe a sequence of test steps to be 

assurance or a testing team on the right side of the dashed executed. The test tool 304 also logs the test results and 

line. It is axiomatic that the development team has extensive progress into log files 302 for further analysis. 



09/03/2003, EAST Version: 1.04.0000 



6,067,639 

7 8 

Script oriented test tool 304, typical of current test the application program during development. By so embed- 

procedures, is external to application program 306. Due to ding test operations within the application program, the 

this limitation, script oriented test tool 304 can only apply development team (with the aid and consultation of the test 

external stimuli that are expected, and accepted, as external team) determines appropriate functions and operations to be 

user inputs by the application user program 306. No inter- 5 tested. In addition, the development team determines the 

mediate levels of functionality testing below that which is expected proper response for each identified test operation 

made visible to the user interface is possible with this embedded within the application program. The testing tools 

external testing approach. Similarly, script oriented test tool and methods of the present invention further provide a 

304 can only detect responses from application program 306 standardized, easy to use, graphical, user interface for the 

that are made externally visible by the application program. 10 performance of test operations on the applications program. 

Script oriented test tool 304 is thus limited in its internal By embedding test operations within the application 

knowledge of the structure and design of the application program, complete automation of test procedures can be 

program 306 and limited to only those stimuli and responses achieved both in provision of stimuli by calling functions, 

that are provided to the external user interface of application and in the capture of test results by verifying the return code 

program 306. For example, a typical response of an appli- is value of a called test operation function or by direct assertion 

cation program 306 may be no more than the update of a function calls of the application program to detect failures, 

display screen to a user interface. Such a response may be These automated test procedures may be used throughout 

detected by script oriented test tool 304 only by capturing the development portion of the software life cycle by the 

the screen dump and storing it for subsequent manual review development team or by the test team. This integration of the 

by a test operator. In this manner, automation of the testing 20 testing of the application program with its development 

procedures is made more difficult because the detection of eliminates the distinct division of labor typical of prior 

the application program 306 response is frequently difficult product testing approaches. Knowledge of the program 

to automate. structure is no longer required to effectively and robustly test 

Hand generated testing is a variant of the test tool com- the application program. The development team and the test 

puting environment depicted in FIG. 3. In a hand generated 25 team may act in concert as a unified group to assure quality 

test environment, a test engineer hand generates a desired in the application program product, 

external stimulus rather than receiving a stimulus from a FIG. 2 depicts a software development and test process 

pre-defined script file 300. Such hand generated testing which may utilize the tools and methods of the present 

limits the reproducability and automation of the test proce- invention. Unlike the process depicted in FIG. 1, there is no 

dures in terms of the generation of external stimuli. The test 30 vertical dashed line indicating the separate responsibilities 

engineer must be certain that all steps which lead up to of the development team and those of the quality assurance 

production of a problem are noted so that the problem may or testing team. Since there is no longer the need for strict 

be reproduced. So called "macro" languages are yet another division of responsibilities under the methods of the present 

variant of the test procedure depicted in FIG. 3. Macro invention, the two teams are referred to under the methods 

language test tools are similar to the script oriented test tool 35 of the present invention as the product development group. 

304 but with reduced capabilities for detecting and recording The product development group, by operation of elements 

the response from application program 306. In essence, 200 and 202, defines the requirement specifications for the 

macro language test tools provide a shorthand notation for application program and develops appropriate program and 

defining a test script of external stimuli to be applied in data structures to implement those requirements. In element 

sequence. 40 202, the product development group embeds test operation 

The well known application program development test object definitions within the program structure of the appli- 
cycle and associated tools depicted in FIGS. 1 and 3, cation program under development. These test operations 
therefore, have several weaknesses as compared to the object definitions are derived from object class definitions 
techniques disclosed by the present invention. Specifically, provided by the tools and methods of the present invention, 
script oriented test tool 304 is only capable of testing those 45 By embedding the test operation object definitions within 
functions made externally visible by the application program the application program (in element 202), the product devel- 
306. Similarly, detection of responses generated by appli- opment group aids in the testing of the application program 
cation program 306 is limited to those responses made by providing interfaces for testing function ality not other- 
visible by the application program. Some such responses wise visible external to the application program structure, 
may be difficult to capture or detect in any automated way 50 This embedding of test operation objects allows automation 
by script oriented test tool 304. An additional problem with of the testing procedure for generating test stimuli as well as 
the script oriented test tool 304, of FIG. 3, arises in the need automated capture and analysis of the test results. In 
to port the test tool 304 to each new computing environment addition, the embedding of the test object definitions permits 
308. The application program 306 development team nor- a broader range of test functionality to be provided as 
mally ports the application program 306 to several different 55 compared to the external testing approach depicted in FIGS, 
computing environments 308. In addition, either the quality 1 and 3. By operation of elements 204 and 206, the product 
assurance or testing team or the development team must port development group performs regression testing of the appli- 
the script oriented test tool 304 and/or the script files 300 to cation program by use of the test tools and methods of the 
the same collection of computing environments 308 inde- present invention as embedded within the application pro- 
pendent of the efforts to port the application program 306. 60 gram by the test operation object definitions. In elements 
This imposes an additional work load upon the product 204 and 206, the product development group determines 
development and test in addition to the normal load for whether too many problems are revealed through a verifi- 
developing the product and identifying problems in the cation and regression testing. If too many problems are so 
application program 306. revealed, the product development group repeats the steps of 
OVERVIEW — PRESENT INVENTION 65 elements 202-206. The product development group next 

The testing tools and methods of the present invention tests the application in elements 208 and 210 to determine if 

involve embedding of test operations within the structure of too many new problems have arisen in the current version of 
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the application program. If the application program is deter- the testing tools of the present invention. A table is built 
mined to be sufficiently problem free by operation of ele- within the test tools of the present invention in which each 
ment 210, then the program is released to application users entry corresponds to one of the defined test operation 
as duplicated in element 214. In either case, all newly objects. As discussed below in further detail, grouping of 
discovered problems are reported for the development group 5 related objects and weighting of the various objects are 
to consider as shown in element 212. The reported open utilized in the automated testing procedures to select appro- 
problems returned to the product development group to priate tests to be invoked. 

repeat the program development cycle in hopes of repairing Element 504 determines whether the user has requested 

the problems. that testing begin immediately or be deferred until some 

The tools and methods of the present invention permit the to subsequent time in the processing of the application pro- 
shift in testing paradigm depicted as between FIGS. 1 and 2. gram. If the testing is to begin immediately, processing 
Whereas under prior approaches all testing, regression continues below with element 510. If in the alternative, 
testing, as well as new testing, was the responsibility of a testing is deferred until some subsequent time, processing 
distinct quality assurance and testing team, the tools and continues with elements 506 and 508 repetitively until it is 
methods of the present invention permit highly automated 15 determined by element 508 that the user has requested the 
testing to be performed at any time in the application commencement of testing procedures. Until that time, de- 
program development lifecycle. ment 506 represents all normal processing of the application 

The tools and methods of the present invention are shown program performing its intended function. The delermina- 

in a block diagram of FIG. 4. Application program 404 is tion of whether to begin testing immediately or to defer until 

operable within computing environment 408. Embedded 20 a later, user specified, time is a matter of design choice. One 

within application program 404 are test operation objects of ordinary skill in the art will recognize the potential value 

406 adapted to permit automated testing of functions within in deferring the start of testing until a particular state of the 

application program 404. The test operation objects 406 are application program is created (by manual operation of the 

derived (by the application programmer developers) from an program). Many equivalent design choices may be made by 

object class provided in a programming library of the 25 those practicing the present invention to implement a desired 

present invention. The library is provided in the form of a delay in the commencement of a test sequence, 

dynamic link library (DLL) to be invoked by application When testing procedures are begun, elements 510 and 514 

program 404. The test operations object library 406 receives are operable to determine which of three possible modes of 

pre-recorded pseudo-random test suite files or playback files testing operation are to be performed. The test tools of the 

400, and logs all testing procedures to playback files and 30 present invention may be operated in a random auto -test 

results to log files 402. As shown in FIG. 4, the tools and mode, a playback auto-test mode, or a manual playback 

methods of the present invention embed the test operations creation mode. Element 510 is operable to determine if 

406 within the application program 404 at the time of its random auto-test mode has been selected. If so, element 512 

development by the product development group. By embed- is next operable to perform the random auto-test procedures, 

ding the test operation objects library (DLL) 406 within 35 If not, then element 514 is operable to determine if the 

application program for 404, the developers may expose playback auto-test mode has been selected. If so, element 

more functionality for testing than is otherwise possible in 516 is operable to perform playback auto-test operations, 

prior test methods which exercise only externally visible Otherwise, element 518 is operable to perform manual 

user interface functions. In addition, the results of each test playback creation procedures. Once begun, elements 512, 

operation may be automatically verified by validation of the 40 516, and 518 are operable until completion of the testing 

return code value from the invocation of each test operation procedures. 

object. An additional benefit of the test tools and methods of FIG. 6A depicts a flowchart of the detailed operation of 

the present invention is derived from the fact that porting of random auto-test element 512 of FIG. 5. In the random 

the application program to a particular computing environ- auto-test mode, test operation objects are randomly selected 

ment 408 automatically performs the required porting of the 45 for invocation to test the application program. The user may 

test operations embedded within the application program. provide weighting values to increase the frequency of cer- 

There is no additional effort required to port testing tools to tain groups of tests being selected as compared to other 

each computing environment along with the application groups of tests, as will be discussed in additional detail 

pro gra m itself. _ , below. Element 600 is operable to prompt the user for input 

TESiFTOOL OPERATIONS y 50 used to adjust the default weighting values applied to each 

FIG. 5 is a flowchart of the~bperation of an application test operation object. Based on the default weighting values, 

program which has embedded the test tools and methods of plus any adjustments entered by the user in operation of 

the present invention. Element 500 represents the normal element 600, element 602 is next operable to generate 

initialization procedures for the application program. These weighted selection tables used for randomly selecting each 

operations include whatever data and code initialization is 55 of the test operation objects according to the weighting 

required to properly perform the intended function of the provided by the user. 

application program. Element 502 is next operable to instan- A weighting table specifies a range of random number 

tiate all the test operation objects defined by the product values which correspond to each entry in the table. The 

development group (and derived from the object class ranges of possible random numbers is equal to the total of 

definition provided by the DLL). The instantiation of the test 60 the weight values for all entries in the corresponding table, 

operation objects involves creating the objects and the ^A weighting table is first generated for the current group 

associated interface functions and making^m'TSowh to weight values to determine which group will be selected, 

the test tools which actually perform tne testing sequences. v When selecting a test operation object, a first random 

In addition, as discussed below, element 502 is operable to number is generated in the range of zero through the total of 

define groups of related objects for simplifying the user 65 all group weights (minus one). The group whose assigned 

interface to the test tool. Instantiating each _test operation range of numbers encompass the randomly generated num- 
object provides the definitionof the objects4o.be Jestedby ber is then selected. After the group is randomly selected, a 
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weighted selection table is generated for the test operation 
objects according to the current weight values for test 
operation objects in the selected group. A second random 
number is then generated to select from the test operation 
objects defined as members of the selected group. The test 
operation object whose assigned range of numbers encom- 
pass the second randomly generated number is then selected. 
For example, if test objects "A" and "B" are in group "X" 
and test objects "C and "D" are in group "Y", and the object 
and group weights are as follows: 



ObjectfCrroup 



Group Weight 



Object Weight 



X 
Y 
A 
B 
C 
D 



60 
30 



20 
10 
100 
150 



Then the group weighted selection table would reflect the 
group weights as follows: 



Group 



Random Number Range 



X 
Y 



0-59 
60-S9 



The test operations weighted selection table if group X were 
randomly selected would appear as follows: 



Group X Object 



Random Number Range 



0-19 
20-29 



The test operations weighted selection table if group Y were 
randomly selected would appear , as follows: 



Group X Object 



Random Number Range 



C 
D 



0-99 
100-249 



Element 604 is next to operable to prompt the user to enter 
a random seed value. The random selection in the random 
auto-test mode is performed via pseudo-random number 
generation techniques. A sequence of "random*' numbers 
generated by such a pseudo-random number generator tech- 
nique may be repeated if the random number generator is 
provided with the same seed value. Such techniques are well 
known to those of ordinary skill in the art and need not be 
discussed further. In order to assure reproducability of any 
problems found, the random number seed value used to 
previously produce the problem may be manually reentered 
by the user so as to reproduce an identical "random" 
sequence of test operations. Element 606 is next operable to 
prompt the user for input parameter values indicating the 
threshold for the number of errors allowed and the total 
count for the number of test operations to be performed. The 
threshold and count values are used to determine the length 
of time the test procedure will be run and a threshold limit 
at which testing will be halted. If the number of problems 



12 



15 



20 



revealed is below the threshold value testing will proceed, 
otherwise testing is halted when the number of errors 
exceeds the threshold value. Similarly, the total test count 
value is used to determine the number of random test 
5 operation objects to be invoked in the random auto-test 
mode. 

Elements 608-626 are performed repetitively until testing 
procedures are baited. Element 608 is operable to determine 
whether the specified error threshold has been exceeded. If 
10 so, testing operations are halted and the test function has 
completed. If not, element 610 is next operable to determine 
whether the total count of test operations has been 
exhausted. If so, testing is completed. If not, processing 
continues by selecting another test operation object invoca- 
tion. 

For each test operation object invocation, elements 
612-626 perform the actual test operation. Element 612 is 
first operable to adjust the weighted selection table for any 
changes in the weighting values. Changes in the weighting 
values are the means by which rules, discussed below in 
additional detail, may alter the testing strategy as the test 
process proceeds. Element 614 is next operable to generate 
a pseudo-random number value for use in the weighted 
selection table to select the next test operation object. The 
pseudo-random number value may be used as an index value 
into the weighted selection tables as described above, or 
otherwise utilized with data structures, well known by those 
of ordinarily skill in the art, in a manner indicative of the 
weighted selections. Element 616 is next operable to record 
the selected test operation object in a playback file. A 
playback file may be utilized in a subsequent test procedure 
to reproduce a problem revealed in random test sequencing. 
The selected test operation object is recorded in the playback 
files before the test is performed to assure that the playback 
file records each test in sequence before the test operation 
execution potentially hangs the application program 
(indicative of a problem in the application program). The 
test procedures and methods of the present invention also 
permit system exception handling to be intercepted by the 
test methods to detect unexpected results in the test opera- 
tion objecUnvocation. This capability of the present inven- 
tion depends upon the computing environment capabilities 
on which the application program (with the test methods 
embedded) is run. 

Element 618 is next operable to invoke the execute 
interface function of the selected test operation object, in 
order to execute the test procedures defined^b^the^producl 
development group corresponding to this selBeted'test opera - 
Uon^jectr-The*execute-funclioncis defined Sy the^product 
d&Msiap niejU group to perform whatever functional process- 
ings-appropriate to tesT. the~desired function corresponding 
to^the^teTt^operation object-Details of an execute function 
are discus sed b elow,with respect to exemplary test operation 
objects. The result's of the executionjnterface function is 
55 recoTded^by-op eratio n pfj&lement 620 inra-result log file. An 
^xecate'interface function may-radicate its success or failure 
by returning a specific result code as the function value ^by*r- 
calling an assertion function in the library methods of the^ 
presentiinvenUoh-to-indicate-an-en-Qneous^test result. The^ 
result»lo£v£le?ma 3 Vjt^ inspected later to determine whether 
theMest sequence^fajled or succeed ed,' v andjn addition, may 
be" usedto" determine at^which step in the test procedures, as 
indicated in the^playback "file,* the test sequence failed. 
Element IS^^is-operable^ to determine whether the test 
execute function result represents a successful or failed 
invocation ofthe Execute interface function. If the execute 
interface function invocation resulted in an error, element 



25 



30 



35 



40 



50 



60 



65 
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624 is next operable to increment the error counter. If the 
execute interface function resulted in a successful 
completion, then the error counter is not incremented and 
operation of element 624 is skipped. Element 626 is finally 
operable to' decrement the total test count. The test process 5 
then continues by looping back to elements 608 and 610 
which verify that test processing should continue or should 
be halted based on the error count and the total test count 
remaining. 

FIG. 7 depicts a flowchart providing additional details of 10 
the operation of elements 516 of FIG. 5, which performs 
playback auto-test mode processing. Element 700 of FIG. 7 
is first operable to prompt the user to enter the name of a 
previously recorded playback file. The playback file con- 
tains information-used by the methods o£the.present inven- 15 
tion to reproduce^previously recorded test sequence. The 
playback auto-test mode.is .most typ icaUy«used»tQ>repioduce 
a problem" previously producecf in a sequence of test steps 
(such as the pseudo-random auto-test mode or manual 
playback creati orugaode disc ussed above) and recordedin^a- 20 
playback file. Test sequence^^producaoility i£ key to the 
product development group in locating and, repairing^ pro- 
gramrmng errors "(bugs) "causing the reported problem. 

Elements 704-710 are next operable repetitively until 
testing procedures have been completed. Element 704 is 25 
operable to determine whether additional test operation 
objects remain to be processed from trie playback file. Each 
entry in the playback file represents the invoc ation o f a 
selected test operation object. ^e^re^7 f 04ithen wdetermines 
wh^ttfeT*additional^ntries are yet to be processed in a 30 
s ekctg d^plavback file" JLT5b^furml*~te^ 
re main u nprocessed, processing of the pi ayb ack a u t o - test . 
mo^e^ incompletes ^d\tesu*ng.haltsC If additional* entries 
rem^m*inT the ~currently,selected playback file^ element 706" 
is pexfoperable^to invoke the execute interface function to 35 
' performjg^^S?lselected test operation. 
" Element 708 is operable to determine whether the result 
of the previous execute interface function indicated success 
or failure. If the execute interface function returned a 
successful completion code, processing continues by loop- 40 
ing back to element 704 to thereby continue the playback } 
auto-test mode by selecting the next test operation from the 
playback file. Conversely, if the execute interface function 
invocation indicates an erroneous result, element 710 is next 
operable to record the erroneous result in a results log file. 45 
Processing then continues by looping back to element 704 to 
complete the playback auto-test mode processing of the 
selected playback file. 

FIG. 8 is a flowchart providing additional detail of the 
operation of manual playback creation mode element 518 of 50 
FIG. 5. Element 800 of FIG. 8 is first operable to prompt the 
user to select a particular pre-existing playback file (for 
editing) or to create a new playback file. Next, element 802 
prompts the user to identify a test operation object from a 
menu of all defined and instantiated test operation objects. 55 
Element 804 is next operable to add the selected test 
operation object to the current playback file (alternatively, 
the user may identify a test operation object to be deleted 
from the playback file, to be edited, or to be moved to 
another position in the playback file). Element 806 deter- 60 
mines whether the user wishes to further edit the playback 
file or whether all changes are done. If further changes are 
desired, execution continues by looping back to element 
802. If no further changes are desired, the method completes 
with processing by element 808 to save the new playback 65 
file to mass storage associated with the present invention. 
TEST OPERATION OBJECTS 



Test operation objects are objects derived from a standard 
C++ class definition provided by the library (DLL). The 
object class may be used by the product development group 
to derive specific definitions and instantiations of test opera- 
tion objects. The test tools of the present invention define a 
minimum of four interface function methods for use by the 
testing tools in performing the required test operation. The 
product development group members define their own test 
operation object class which implements at a minimum the 
execute interface methods described above. The other three 
of the four basic interface methods are provided with default 
definitions in the object class. The programmer may use the 
standard object definitions of the three other interface 
methods, or may replace them (override them) as appropri- 
ate for the particular objects test goals. The specific func- 
tionality within each of the four basic interface functions is 
provided by the product development group engineer to 
actually perform the required test operation and return a 
success or failure result. The basic four interface functions 
are: execution, data input, file and memory I/O, and rules. 

The execute interface function must be provided in the 
derived class object. There is no default implementation of 
an execute function because the execute function contains 
all of the code necessary to perform a particular test opera- 
tion. The execute interface function performs only the 
precise function calls required to execute the desired test 
operation function. Any input or output requirements are 
handled through the data input or file and memory I/O 
interface functions. The following is a simple example of an 
execute function used to perform an open file operation. 



DWORD COpcnFUc;:Exccutc( ) 



{ 



// Log will write the provided string to the logfile and return FALSE 
// if the operation should skip execution, 
if (Log(m_strOperationName)) 



{ 
} 

return Error ( ); // Return success or failure. 



GetApplication( )->OpenFfle(m__strFileName); 



This exemplary execute function (the Execute function of 
the COpenFile test operation object) simulates a users 
selection of a file/open menu item exactly as though the user 
has selected the file/open menu item by operation of the 
application program. 

At least two functions may be defined as part of the data 
input interface method. These functions are invoked indi- 
rectly by the invocation of the execute interface function 
when data input is required. A first function in the data input 
method can automatically generate a desired input value. A 
second function in the data input interface method allows 
manual input of a data value by prompting the test user. The 
following are example functions implementing a typical 
automatic generation, and manual, data input function. 



BOOL COpenFile: <jcnerate( ) 
{ 

CFUcUst •pFfleList - NULL; 

// Ask the application for a list of Mies with a certain extension. 
pDUeList » GetAppIication( )->FindFdes("".t2ct"); 
// If there arc none, then fail, 
if (pFilelist( >>Empty( )) 
return FALSE; 
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Each of the groups denned by the development engineer 
-continued may be associated with a weighting value. Weighting values 

~~ are used in conjunction with the pseudo-random auto-test 

EZ^^Z&S*** ft mode described *"* to >he frequency distribution of 



//Get the name of the picked file and save it. ' 5 random calls lo the various test operaUGArobjects. 

m„strFileName - pFileUst( >>GetFileName(nPickedFile); Wjj jghtingtfgalufc^^ I^aHve humbeiMaf 



return TRUE; calls to test operation objects within one. group as distinct 

n v. from the number~6f calls to test operation objects associated 

I -with-another. group. Jn other words, the development engi- 

crueDiaiog digFOeDiaiog; 10 neer may define one group to be "randomly" selected more 

bool bRct = false; fremoentfyjLhajiJs-another group. In addition to the group 

I weighted further to determine its relative frequency of 

// if the usei dicks *OK' save the new file name, invocation ^compared to each other test operation object in 

// otherwise FALSE will be returned and the edit/create 15 tJj e same group* 

S.»^,<,.«,« ta «(); EMBEDDWO-WSTJBBJECTS. - " 

bRet - TRUE; The test^o^jcets.dcfificxl.by^bc^TOduet 1 development team 

} nlu^fbedefined and instantiated in the initialization code of 

return bRet the. application program as shown in element 502 of FIG. 5. 

20 The following C++ code ^sample is exemplary of an 

approach to embedding the test objectjdefimtion and instan- 

In general, the Generate function is used by the testing tiatioQ m mc application program. In th^foUowng^exem^ 

tools to create pseudo-random test data input values. Other plary thc test melhods m6 structures of the present 

methods besides random generation may be used in a mvC ntion are referred to by the Microsoft trade name 

Generate function to automatically create appropriate test 25 TestWizard. It is presumed in the exemplary code below that 

vahies For example sequencing through a variety of test ^ a p prop riate header file(s) have been included to provide 

values from low to high may be appropriate m some test .* • » «• . * } n 

™ , . j 1 11 j • the required object class definitions, 

cases. The automatically generated values are also logged in — i j 

the playback file so that any problems revealed by the test 

sequence with specific input values may be easily repro- 3Q 

duced. a Q Jcaic ^ sufo object which defines the parameters 

The exemplary Manual input function presents exemplary 0 f the pseudo-random 

code used to display a dialog box in which a test user of the // auto-test mode of operation. This object defines the groups 

application may enter data input values to proceed in the test of t*st operations 



process. As in the Generate exemplary function, any data " ^ the native weights among the groups and opw^iona 

input value received is logged in the playback file so that a * ' ncw CTestSuuc(MAX_GROUP > max_operatton, 

test sequence may be easily reproduced. f{ c rcatc the TestWizard object - the heart of the methods and 

The file and memory I/O function interface method per- structure of the 
mits simulation of input data values retrieved from tempo- // present invention, 
rary files or memory buffers. Such file or buffered I/O m_4>TestWizard - new ciestwizard; 

requests generated by invocation of the execute function of 40 '' five groups and their respective weights in the test suite object 

0 . . . . // (initially default values are used for the group weights) in the test 

the test operation object are satisfied by simulating the 5^ object, 
writing and reading of values to and from files or memory // The five groups are: 
buffers by the application program. By simulating these H creategroup 
operations, the values used may be captured and thereby ^ MANffSioup 

reproduced in later invocations of the playback file (i.e. to 45 // lavergroup 
reproduce a test sequence which resulted in a failure). // selecttongroup 

The rules interface method comprises a set of functions m_pTestSuitc ->SctGroupWeight(CREATEGROUP, 
which determine the propriety of running a particular test j^J s^^L P ^i^iLEGKOV? f default__weight); 
operation function, or group of test operation functions in m _pTestSuite->SetGroupWeight(MANiPGROUP, 

light of the current context of the application program, or the 50 default_weighT); 
context of the test processing. The rules interface method is m_pTestSuite->SctGrou P Weight(LAYERGROUP f 

DEFAULT WEIGHT)* 

CSS»»^^^^!!L$^. m^TestSulu^SetGripWeightCSELECnONGROUP, 



f^scuss^^elo^m,addiUonaLdetaU. m ^TestSu1te->SetGrc 

J^^^ 0fiER ^62J^i^ CT ^|^^^ > y DEFAULT_WEIGHT); 

P^As a convenience to me^^e^in^useffof an application J m_pTestWizard->SetT 



urogram w^th ihe^tes^to^ according to the^ 

present invention, functions»are prov*icle*8 • i mnhe3^Lopefa^ 



iUSer^ pf an application ± m_pTestWizard->Set1estSuite(m_pTe6tSuite ) ".sui", ".wiz"); 

.55 m_pTestWizard->SetApplicationName("DcmoApp"); 



m_pTestWi2ard- >SetPIaybackEx tension (**.wiz^; 

~ - // Get access to the TestWizard User Interface object 

ttion objecHibrary (DLL) of the present' invention to permit m_j)Tesiwizard->GetTestwizaidui (ftptestwizui); 

a development engineer to associate related test operation // Define the pages to be displayed in the ui along with the 

obiec^nto^grauus!! . Function calls in the DLL permit a options page 

te^e^e^r to create new ^up^finitions. Each 60 *£%£%£^£S£$£^ 

group becomerassociated with a uniime'group identifier^ pt estwizUi->AddPage(new CManipuiatioaPageCptestwizUi)); 

textual name may be associated wim.e^a^di-group-fof'use in ptestWizUi->AddPage(new CLayeiPage(ptestwizUi)); 

prompting a test user to enter parameters associated with the ptestWizUi->AddPage(new cseiection(ptestWizU0); 

group. Function calls-within the DLL used to defineeacb test " ^AlT^T*™^!" ^ ^JT^ZX opemtfa " ^ 

° r . , . . t , r . . ... // NEWOP (use the CNewFile test operation object) 

operation object include, a parameter^to identify the pre- 65 n openop (Not Yet implemented) 

defined group with which*the test operation object is -to be // CLOSEOP (use the ccioseRJe test operation object) 

associated. 
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ment and utilize the features of the present invention. Many 

-continued assumptions regarding variable types and scopes are made in 

— — — — — _ _ — the above code segments. One of ordinary skill in the art will 

// saveop (use the csavcFDc test operation object) readily recognize the intended meaning of the code sample, 

newop. 5 P-do-code suggestive of the intended 

new fsxCTCHNEWGROUP) us& °f tDe methods and structures of the present invention. 

CNewrae(injiTbstwi»nO> f PSEUDO-RANDOM AUTO-TEST MODE USER INTCR- 

m_pTesiSuite->SetOperatiop(nUEGROUP > OPENOP, new FACE 

CN^Iopcratio n(m_pTcstWizflrd)) ; FI G . 9 is an exemplary display screen used in the testtools 

^^ l |S^% C nn^ LEGR0Un CLOSEOP ' m and methods of the P rcsent invention IcTpeTmTthelest^ user 

new (SKETCHNEWGROUP) 10 , . , . r r 

caoseFac(m_pTestwi2ard)); to*denne~thew^ighUvalues~iised _foff^each- test operation 

m_pTcstSuite->sctoperation(FiLEGROUP, saveop, ob ject and for each group of test operation objects. The set 

new (sketchnewgroup) of test operatiolfobjects andtheir groupings are defined by 

rn^^^Z^lEOKO^ SAVEASOP, new ^f?*V ™ 

CWIopcration(m__pTc S tWizard)); 15 ded in the source code of the application program (as 

// Invoke the UI to display the user interface and begin testing. discussed above with respect to FIGS. 2 and 4. Label 900 of 

m_pTestWizard->lnvokeUl( ); FIG. 9, discussed in detail below with respect to FIG. 10, 
^ indicates the hidden "OPTIONS** display screen used to set 

The above, exemplary initialization code may be inserted g loDal °P tion values for me pseudo-random auto-test mode 

in the application program at any desirable position in the 20 operation. In FIG. 9, four exemplary groups have been 

initialization of the application. The invocation of the defined as indicated by the labels 902-910. Specifically the 

TestWizard User Interface (to commence testing operations) groups are named: "CREATION/DELETION" (label 902), 

may be inserted in the application at any desirable position "FILE" (label 904), "MANIPULATION** (label 906), 

including: at startup of the application as shown above, "LAYER** (label 908), AND "SELECTION*' (label 910). 

following operation of some particular function, as an option 25 The display screen of FIG. 9 is an exemplary screen for 

in the standard user interface of the application (i.e. in a entry of weighting values by the test user to be associated 

menu item of the application program), or at any other with the test group named "CREAIlQ^/DEffiTlOTr^The 

desirable point in the operation of the program. g^P- weighty ^yalueentered in field.912setsthe group-weight 

The SetOperation function calls specify the test operation iqrJhis test operation group relative to all otfiertesropenr- 

object (third parameter) to be associated with the group and 30 tibn groups defined by the product development group. For 

operation specified by the first two parameters. The test example, if the other four groups (labeled "FILE", 

operation objects are derived from a COPERATION test "MANIPULATION*', "LAYER**, and "SELECTION**) have 

operation object class defined by the methods and structures g^p weights of 75, 25, 100, and 125, respectively, then the 

of the present invention (and included by appropriate header test operations in this test group ("CREATION/ 

files in the exemplary code above). The CNewFile, 35 DELETION**) will be selected (50/(50+75+25+100+125)) 

CCloseFile, and CSaveFile test operation objects are derived time s or 13.33% of the pseudo-random selections, 

from the COperation object class. The CNYIOperation Within each test group, each test operation may be indi- 

subclass of the COPERATION class is defined by the vidually weighted and enabled/disabled. In the 

methods and structures of the present invention to provide a " CREATION/DELETION" stest group of FIG. 9, there are 

default test operation object as a place holder for functions 40 six test operations labeled: "CREATE", "DUPLICATE", 

not yet implemented by the development group. "PASTE", "CUT", "COPY**, and "DELETE." These six test 

A COPE&ATION class object may be derived from the operations may be individually enabled or disabled by 

base class in accord with the following C++ code sample: checking or clearing the associated ,check box, 954-964, 

CNewFile: :CNewFile(CTestWizard*pTestWizard) J"*^ 1 * 1 / 1 ^t™, each of these ? ix test operation may 

:COperation(pTestWizard) 45 be individually weighted by catering a weight value in the 

r associated weight field, 814-924, respectively. The enabled 

* test operations within the group will be pseudo-randomly 

//Body of code to implement a new file operation selected according to the ratio of their respective weights 

} whenever their containing group, "CREATION/ 

The execute interface method function must be imple- 5Q DELETION", is selected. In other words, the "CREATE** 

mented in every derived object. A skeletal exemplary test operation will be selected 50/(50+10+25) times, or 

execute function might appear as follows: 58.82% of the times when the "CREATION/DELETION" 



DWORD CNewFile::Execute( ) 
{ 

CString strMessage; 

str Message. Fonnat( M FUe New Called\n"); 
if(Log(strMessage)) 

AfxGelMainWnd( )->SendMessage(WM_COMMAND, ID_FIUE_NEW); 
return Error( ); 

} 



One of ordinary skill in the art will recognize that the 65 
above exemplary code segments are intended only as group is selected. Since the weighting of the group is 
demonstrative of the structure of actual code used to imple- multiplied by the test operation weight, the "CREATE" test 
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operation will be selected 7.84% of the lime among all all test operations are invoked normally. This feature is 

defined and enabled test operation objects. useful to proceed beyond known problems in a test sequence 

Button fields 926-932 are used to save and retrieve the to reveal other new problems. This "persistent" error infor- 

values and settings entered in the above discussed fields mation is retained and accumulated through all test opera- 

(912-924 and 954-964) for future reference. Use of buttons 5 tion invocations and through a plurality of test sequence 

such as these are well known to those of ordinary skill in the runs. Field 1024 is a button which, when selected 

art and need not be discussed further. ("clicked'*) by the user, will clear the persistent log of 

Button field 934 begins the pseudo-random auto-test erroneous results to again permit testing of all operations in 

mode operations as defined by the parameters entered in the a test sequence of the pseudo-random auto-test mode, 

above discussed fields (912-924 and 954-964). FIGS. 5-8, to Field 1026 is used to enter a maximum group/operation 

discussed above, present the detailed operation of the weight value. This maximum value is used to automatically 

pseudo-random auto-test mode. generate weighted values for all groups and for all opera- 

FIG. 10 is an exemplary display screen used to prompt the tions within a group when button 1028 is clicked, 

test user of the present invention to enter parameters Button fields 1030-1036 are used to save and retrieve the 

(options) used in the pseudo-random auto-test mode of 15 values and settings entered in the above discussed fields 

operation. Label 900 of FIG. 10 indicates the semantic use (1012-1028) for future reference. Use of buttons such as 

of the screen display for entry of "OPTIONS" in the running these are well known to those of ordinary skill in the art and 

of the pseudo-random auto-test mode. Labels 902-910, as need not be discussed further. 

discussed above with re fere nee to FIG. 9, indicate the names Button field 1038 begins the pseudo-random auto-test 

of exemplary groups of test operation objects as defined by 20 mode operations as defined by the parameters entered in the 

the development. above discussed fields (1012-1028). FIGS. 5-£, discussed 

Fields on the "OPTIONS" screen of FIG. 10 are used to above, present the detailed operation of the pseudo-random 

define parameters which control the operation of the pseudo- auto-test mode. 

random auto-test mode of the present invention. Field 1012 PLAYBACK FILE EDITING USER INTERFACE 
is used to enter the overall error or failure threshold value. 25 FIG. 11 is an exemplary display screen designed to permit 
This value determines the maximum number of erroneous a test user to edit the contents of a previously recorded 
results permitted of all test operation invocations before the playback file. In the playback file creation mode of 
pseudo-random auto-test mode is terminated. Use of this operation, a test user edit a previously recorded playback file 
overall threshold value is discussed above with respect to to customize the test sequence, or create a new test sequence, 
element 608 of FIG. 6 A and elements 622 and 624 of FIG. 30 In the screen display of FIG. U, a list of available test 
6B. Field 1014 of FIG. 10 is used to enter a single failure operations 1100 is positioned on the left side of the screen, 
threshold value for use in the methods of the present A list of the test operations currently selected for the test 
invention as discussed above with reference to FIGS. 5-8. sequence, the current script 1102, appears on the right side 
The single failure threshold value is used to determine of the screen. A user may highlight ("click" on) a desired test 
whether any particular test operation has exceeded this 35 operation from the list of available test operations, then click 
failure threshold. The test tools and methods of the present on the "ADD" button 1104 to add the highlighted test 
invention account for the number of test operation failures operation to the current playback 1102. Conversely, the user 
associated with the invocation of each test operation object may highlight a test operation in the current script 1102 and 
as well as the total number of test failures. This threshold click the "REMOVE" button 1106 to remove the highlighted 
value is used in a manner similar to that of the overall test 40 test operation from the current script 1102. 
failure threshold to terminate the invocation of a particular In addition to the "ADD" and "REMOVE" functions, a 
test operation object in the pseudo-random auto-test mode of highlighted test operation in the current playback 1102 may 
operation. As shown in FIG. 10, threshold values of zero (or be skipped by clicking the "DISABLE" button 1108. A 
less) will terminate the pseudo-random auto test mode (field pause in the test sequence may be added to the current script 
1012) or the continued invocation of a particular test (field 45 by highlighting a test operation in the current script 1102 and 
1014) as soon as any failure occurs in the invocation of a test clicking the "BREAK" button 1110. This marks the high- 
operation object. lighted test operation such that the operation may pause 

Field 1016 is used to enter the random number generator before executing, 

seed value for the start of the pseudo-random test selection. Any test operation in the current playback 1102 may be 

To reproduce a particular "random" test sequence, the user 50 altered by highlighting the test operation and clicking on the 

may enter the same seed value as used in a previous run of "EDIT* button 1112. 

that test sequence. Field 1018 is used to enter the total Test operations in the current list 1102 may be moved up 

number of test operation objects to be invoked for the or down in the current list 1102 relative to other test 

pseudo -random test sequence. The pseudo-random auto-test operations by highlighting the test operation to be moved 

mode will terminate operation after this number of test 55 and then clicking on the "MOVE UP' button 1114 or 

operation objects are invoked (regardless of the success or "MOVE DOWN" button 1116. 

failure of the test operation). Field 1020 specifies the direc- Button fields 1118-1126 are used to save and retrieve the 

tory on a mass storage device associated with the computing values and settings entered in the above discussed fields 

environment running the test. Files used in the operation of (1100-1102) for future reference. Use of buttons such as 

the pseudo-random auto-test mode (playback and result log 60 these are well known to those of ordinary skill in the art and 

files, etc.) will be stored in this directory for later reference. need not be discussed further. 

Check box field 1022 is used to enable the persistent RULES 

storage of single operation threshold values and error counts As shown in FIG. 6B, element 612 is operable to adjust 

from previous invocations of test operations to accumulate the weight selection tables used in pseudo-random auto-test 

the error counters. When field 1022 is checked, test opera- 65 mode for any changed weights. Rules as used herein, are 

tions which previously produced erroneous results exceed- functions which are invoked in association with the invo- 

ing the designated threshold value are skipped. Otherwise, cation of the execute interface function wherein the rules 
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function modify the current weight value for one or more 
test operation objects or groups. By modifying the weight 
value of a particular test operation object or a particular 
group, the rules function may alter the frequency with which 
certain test execute interface functions are invoked. The 
conditions in which the rules function will modify a weight 
value for a selected test operation object or group, may be 
completely defined by the code implementing the rules 
function as written by the product development group. 

Rules, as defined herein, may be used for example, to 
reduce the frequency of a particular test operation depending 
on the current state of operation of the application program, 
or for example, depending upon the current state of progress 
in running the test application program. Changing the fre- 
quency of invocation of a particular test operation object or 
group may include, for example, reducing its frequency of 
invocation to zero (i.e. "not available"). For example, func- 
tions which perform an edit menu copy or paste function 
may be valid only when a file is "open" in the context of the 
application program. Rule functions may be used to assure 
that the execute interface functions for the copy or paste test 
operation objects will never be called until after the test 
operation object execute interface function for the files/open 
menu operation is invoked successfully. 

The rules interface functions for each test operation object 
are invoked to adjust the weighted selection table before the 
next random selection is made in the flowcharts of FIGS. 6A 
and 6B. 

What is claimed is: 

1. In a computer application program operable on a data 
processing system, a method, integrated with said computer 
application program, for testing said computer application 
program comprising the steps of: 

(a) embedding at least one test operation object definition 
within a program structure of said computer application 
program; 

(b) calling a function to instantiate at least one test 
operation object from said at least one test operation 
object definition, wherein said test operation object 
defines a function of said computer application pro- 
gram to be tested, and wherein said test operation 
objects includes an execute interface method; and 

(c) calling said execute interface method defined by said 
test operation object to generate a test result. 

2. The method of claim 1 further comprising the step of 
recording said test result. 

3. The method of claim 1 wherein step (b) instantiates a 
plurality of test operation objects from a plurality of embed- 
ded test operation object definitions each said plurality of 
test operation objects defining a function of said computer 
application program to be tested, and wherein said test 
operation objects each include an execute interface method. 

4. The method of claim 3 wherein step (c) includes the 
step of selecting one of said plurality of test operation 
objects. 

5. The method of claim 4 further comprising the step of 
repeating steps (b) and (c). 

6. The method of claim 5 further comprising the step of 
recording the selected one of said plurality of test operations. 

7. The method of claim 6 wherein said step of selecting 
comprises the step of picking said one of said plurality of 
test operation objects from among a plurality of test opera- 
tion objects previously recorded. 

8. The method of claim 4 wherein said step of selecting 
comprises the step of pseudo-randomly picking said one of 
said plurality of test operation objects from among said 
plurality of test operation objects. 



to 



9. The method of claim 4 wherein each of said plurality 
of test operation objects further includes rule elements 
defining when said each of said plurality of test operation 
objects may be executed relative to others of said plurality 
of test operation objects. 

10. The method of claim 9 wherein said step of selecting 
further comprises the step of applying said rule elements to 
determine that a test operation object may be selected. 

11. In a computer application program operable on a data 
processing system, a method for integrating a computer 
application program with the testing thereof comprising the 
steps of: 

(a) embedding test operation object definitions in said 
computer application program; 

(b) instantiating at least one test operation object defined 
by operation of step (a), wherein said test operation 
object defines a function of said computer application 
program to be tested, and wherein said test operation 
objects includes an execute interface method; 

(c) calling said execute interface method included in said 
test operation object to generate a test result; and 

(d) recording said test result for pass-fail analysis of said 
computer application program. 

12. The method of claim 11 wherein step (b) instantiates 
a plurality of test operation objects each defining a function 
of said computer application program to be tested, and 
wherein said test operation objects each include an execute 
interface method. 

13. The method of claim 12 wherein step (b) includes the 
step of selecting one of said plurality of test operation 
objects. 

14. The method of claim 13 further comprising the step of 
repeating steps (b), (c) and (d). 

15. The method of claim 14 further comprising the step of 
recording the selected one of said plurality of test operations. 

16. The method of claim 15 wherein said step of selecting 
comprises the step of picking said one of said plurality of 
test operation objects from among a plurality of test opera- 
tion objects previously selected and recorded. 

17. The method of claim 13 wherein said step of selecting 
comprises the step of pseudo-randomly picking said one of 
said plurality of test operation objects from among said 
plurality of test operation objects. 

18. The method of claim 13 wherein each of said plurality 
45 of test operation objects further includes rule elements 

defining when said each of said plurality of test operation 
objects may be executed relative to others of said plurality 
of test operation objects. 

19. The method of claim 18 wherein said step of selecting 
further comprises the step of applying said rule elements to 
determine that a test operation object may be selected. 

20. A computer readable storage device, readable by a 
computer, embodying a program of computer instructions 
executable by a computer to perform the method steps for 
integrating testing of the program with execution of the 
program, the method comprising the steps of: 

(a) embedding at least one test operation object definition 
within a program structure of said computer application 
program; 

(b) calling a function to instantiate at least one test 
operation object from said at least one test operation 
object definition, wherein said test operation object 
defines a function of said computer application pro- 
gram to be tested, and wherein said test operation 
objects includes an execute interface method; and 

(c) calling said execute interface method defined by said 
test operation object to generate a test result. 
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21. The storage device of claim 20 wherein the method 
further comprises the step of recording said test result. 

22. The storage device of claim 20 wherein the method 
step (b) instantiates a plurality of test operation objects from 

a plurality of embedded test operation object definitions 5 
each said plurality of test operation objects defining a 
function of said computer application program to be tested, 
and wherein said test operation objects each include an 
execute interface method. 

23. The storage device of claim 22 wherein the method 10 
step (c) includes the step of selecting one of said plurality of 
test operation objects. 

24. The storage device of claim 23 wherein the method 
further comprises the step of repeating steps (b) and (c). 

25. The storage device of claim 24 wherein the method 15 
further comprises the step of recording the selected one of 
said plurality of test operations. 

26. The storage device of claim 25 wherein the method 
step of selecting comprises the step of picking said one of 
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said plurality of test operation objects from among a plu- 
rality of test operation objects previously recorded. 

27. The storage device of claim 23 wherein the method 
step of selecting comprises the step of pseudo- randomly 
picking said one of said plurality of test operation objects 
from among said plurality of test operation objects. 

28. The storage device of claim 23 wherein each of said 
plurality of test operation objects further includes rule 
elements defining when said each of said plurality of test 
operation objects may be executed relative to others of said 
plurality of test operation objects. 

29. The storage device of claim 28 wherein the method 
step of selecting further comprises the step of applying said 
rule elements to determine that a test operation object may 
be selected. 
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